Sql Extensions for Domain Agnostic Data Representational Consistency

The explosion of big data exacerbated the significance of data quality in decision support systems and data warehouses. Data inconsistency, as a significant data quality problem especially for heterogeneous databases, is mainly researched in three aspects: data integrity, semantics, and representational inconsistencies. The data integrity aspect has been well researched and implemented into DBMSs and data warehouses. The methods to detect and resolve semantic and representational inconsistency problems have been developed within a certain context. However, for a general data quality context, there is a lack of methods available for domain agnostic data inconsistency problems. Historically, data representational inconsistency has already been discussed in the pre-processing of data cleansing frameworks and data quality tools. However, since they deal with the problems in a certain context or based on a specific domain, users must obtain specific information about the data such as the master data and the data dependencies in order to address these data inconsistency issues. This thesis focuses on domain agnostic data representational inconsistency problems in a general data quality context in a relational database. In this thesis, we employ a declarative method which introduces SQL extensions instead of writing massive amounts of code. To improve data representational consistency, we propose a user-driven pattern-based framework using the iterative and interactive approach and string pattern matching technology. There are three main subtasks: a) design a complete and nearly mutually exclusive pattern library, b) detect all the possible patterns for each record in the target column, and c) unify the inconsistent data records. Then, we improve the pattern detection algorithms for inconsistent data records through a modified DFA (Deterministic Finite Automaton) and comprehensive experiments are conducted to verify the accuracy and efficiency of the proposed approaches. The evaluation results demonstrated that the proposed methods in this thesis have better performance over the naive solution. Finally, we implement a toolkit based on the proposed framework and methods.